feat(npu): capture AsyncCAM layered FFN in a reusable graph - #418

…sable graph (#436)

## Purpose

Backport #418 to `release/v0.26.0rc2`: capture a reusable AsyncCAM
layered FFN graph for DeepSeek V4 W4A8, with cross-role startup
coordination. Eager FFN remains supported; non-eager FFN requires FULL.
Attention remains eager.

## Changes

Cherry-picked all eight commits from #418, through
`4e86c7182e570c650f98d6c6b45910fe084f99eb`, without conflicts or
additional adaptations. All 16 changed files are byte-for-byte identical
to the source PR head.

The startup coordinator owns mode agreement, paired warmup, readiness,
and failure propagation. Runners own computation and graphs, the
connector owns communication resources, and the worker owns its receive
thread. Native kernels and ABI remain unchanged.

## Validation

- Ruff 0.15.13 lint and format checks passed for all 16 changed files.
- Python compilation and `git diff --check` passed.
- Attempted the three focused unit-test modules: all three were skipped
because the local environment lacks `torch` / `torch_npu`; no runtime
tests executed locally.
- NPU service validation was not rerun for this release branch. See #418
for the original FULL/eager validation results; those results are
source-PR evidence, not validation of this backport.

---------

Signed-off-by: jiangkuaixue123 <jiangxiaozhou111@163.com>

…h Store (#448)

## Purpose

Fix AsyncCAM startup with different Attention and FFN HCCL buffer sizes.
With Attention at 1024 MiB and FFN at 4096 MiB, `_get_store()` fails in
`HcomBroadcast` with inconsistent `inCclBufferSize` / `outCclBufferSize`
metadata. This path runs in eager mode as well as FULL mode.

Exchange the startup namespace through the existing rendezvous Store so
this host metadata no longer requires an HCCL collective.

## Issue

Related to #436, which backported the startup coordinator from #418.
Based on `release/v0.26.0rc2` at `cda849a5`.

## Scope

- In scope: `_get_store()` and focused startup tests; production diff is
+8/-20 lines.
- Out of scope: CAM data transfer, native kernels, inference scheduling,
and recovery using a surviving Store after partial rank restarts.

## Implementation Notes

- Rank 0 writes a random namespace to a bootstrap `PrefixStore`; all
ranks wait up to the existing 300-second startup timeout, read it, and
cache the resulting namespace-prefixed Store.
- This relies on one startup per fresh CAM rendezvous Store.
Cached-context validation and subsequent mode/ready/failure coordination
are preserved.
- Remove the nonce tensor, broadcast, device synchronization, and
device-to-host readback from `_get_store()`. Existing
communication-warmup synchronizations remain in place; Store readiness
is not a device-completion fence.

## Test Plan

- Run `python -m pytest -q --confcutdir=tests
tests/unit/v1/worker/test_async_cam_startup.py` in a torch_npu
environment. New cases use a real HashStore with 16 threads to cover
leader-first/leader-late arrival, cached access, isolation across fresh
Stores, and missing-leader timeout; device nonce operations are
forbidden.
- Run Ruff check/format and `git diff --check` for the two changed
files.
- On the same 16 DIE, compare the previous local Store handshake
workaround with this patch, using 12A+4F, eager mode, A1G/F4G, HTTP
concurrency 16, and identical frozen inputs/routes. Each cold-cache run
contains 16 complete sessions / 328 requests, with one output token per
request.

## Test Result

- Startup unit tests: **12 passed, 0 skipped**. Ruff 0.15.13
check/format and diff check passed.
- Both hardware runs: **328/328 successful**, with matching startup
arguments and performance environment variables. Only this startup
module differs between the deployed packages.
- Cached tokens and actual prefill tokens match exactly: 18,604,032 and
1,904,633 respectively, with no preemptions. Request routing, dependency
order, and concurrency audits passed.

| Metric | Previous Store workaround | This patch | Change |
|---|---:|---:|---:|
| Replay duration | 104.94 s | 105.31 s | +0.36% |
| Throughput | 3.126 requests/s | 3.115 requests/s | -0.36% |
| TTFT P95 | 7.115 s | 7.090 s | -0.36% |

The unpatched HCCL-broadcast version cannot start with this A1G/F4G
configuration, so it is not the performance baseline. This single paired
run shows essentially unchanged performance, not statistical proof of
zero regression. Hardware coverage is limited to eager A1G/F4G; FULL
mode and arbitrary buffer sizes were not revalidated. Natural cache
allocation differed by three blocks in one domain (0.0053%); measured
cache/prefill work was identical.

## Docs Impact

No documentation files changed. The fresh-Store lifecycle assumption is
documented next to the implementation; no configuration option or public
API is added.

---

<details>
<summary>Essential PR Checklist</summary>

- [x] Purpose is clear and linked to public context.
- [x] Scope is bounded.
- [x] Compatibility with vLLM v0.26.0 is considered.
- [x] No changes are made to the vLLM source checkout.
- [x] The change stays in the plugin-owned startup coordinator.
- [x] No compatibility shim or runtime monkey patch is added.
- [x] No new device-dependent import or import-time work is added.
- [x] Validation evidence and hardware coverage limits are included.
- [x] Documentation impact is stated.

</details>

Signed-off-by: jiaran-king <jiaranran2@gmail.com>